<html>
<head>
<link rel=stylesheet href="style.css" type="text/css">
<title>Tutorial - Lustre</title>
</head>

<body>

<body>
<center>
<h1>Tutorial - Lustre</h1>
</center>
<p>
<h3>Introduction</h3>
This tutorial is intended to help you get started monitoring a system on which the
<a href=http://wiki.lustre.org>lustre</a> filesystem has been installed.  This is <i>not</i>
intended to be a tutorial on how to use such a filesystem.  In terms of the basics,
there's really not much to say other then tell collectl to monitor lustre by
specifying an <i>l</i> with <i>-s</i> along with any other subsystems you may want to monitor.
<p>
As you should be aware of by now, collectl will try to display everything you've
selected in <i>brief</i> format if it can, writing all your data out on a single
line for each sampling interval.  
This can get quite wide depending on how many subsystems you choose to monitor and
while you can cerainly specify <i>collectl -s+l</i> to add lustre to the default 
subsystems, the output width is too cumbersome for this tutorial.
Therfore I'll use some other subsystems and switches in the examples
to mix things up, showing that there are a lot of possible combinations.  In this 
first example, we see what happens when you run collectl on a lustre client and
request cpu and memory data along with lustre.

<div class=terminal-wide14><pre>
$ collectl -scml
#<--------CPU--------><-----------Memory----------><-------Lustre Client------>
#cpu sys inter  ctxsw free buff cach inac slab  map  Reads KBRead Writes KBWrite
   0   0   101     26   3G   6M  29M  12M  43M  65M      0      0      0       0
</pre></div>
<p>
Collectl is actually very intelligent about dealing with lustre because if you
were to run the identical command on an OST, it would recognize that too and 
change what it shows accordingly as you can see below.

<div class=terminal-wide14><pre>
$ collectl -scdl
#<--------CPU--------><-----------Disks-----------><--------Lustre OST------->
#cpu sys inter  ctxsw KBRead  Reads  KBWrit Writes KBRead  Reads KBWrit Writes
   0   0   100     28      0      0       0      0      0      0      0      0
</pre></div>

In fact, if you're also running a client on an OST it will show both!
I've also included time stamps to make the output a little more intersting.

<div class=terminal-wide14><pre>
$ collectl -scl -oT
#         <--------CPU--------><--------Lustre OST-------><-------Lustre Client------>
#Time     cpu sys inter  ctxsw KBRead  Reads KBWrit Writes  Reads KBRead Writes KBWrite
14:35:32    0   0   103     24      0      0      0      0      0      0      0       0
14:35:33    0   0   123     53      0      0      0      0      0      0      0       0
</pre></div>

As expected, you can run collectl on a system that has any combination of MDS, OST and 
client services and it will show you what you want to see.  For more detail on some of
the more advanced concepts beyond the scope of this tutorial you can read more about
how collectl deals with Lustre <a href=Lustre.html>here</a>.
<p>
And finally, don't forget any of this data can be written to a file for continuous 
logging, played back later and even converted to a format suitable for plotting.
In fact collectl is configured to monitor lustre by default when run as a daemon
so all you need to do to begin collecting the basic data shown above is
<i>service collectl start</i>.
<p>
<h3>Beyond the Basics</h3>
For many users of collectl, you are now sufficiently equipped to tackle most lustre
monitoring tasks, but for those more exotic situations there's so much more you can
do.  
<p><B>CLIENTS</b><p>
Let's start off by looking more closely at client data.  As with all other collectl
data, one can switch between summary and detail data by simply entering an 
upper case <i>L</i> instead of a lower case one as I've done below.  
The actual content of what is
displayed will depend on whether you are on a client, OSS (there is currently no 
detail data for an MDS), but as you can see for clients, this data is broken down
by filesystem and just to make the display a little more interesting I decided to
show the timestamps in milli-seconds.  Naturally if there is only one filesystem, 
the data with match that displayed in summary mode.

<div class=terminal-wide14><pre>
$ collectl -sL -oTm
# LUSTRE CLIENT DETAIL
#            Filsys   Reads ReadKB  Writes WriteKB
15:10:06.009 spfs1       0      0       0       0
15:10:06.009 spfs2       0      0       0       0
</pre></div>

Just to take it one step futher, it turns out that lustre actually tracks client I/O by
individual OSTs and sometimes that is more interesting, so there is a special mode
for lustre in which you can specific <i>LL</i> which you should think of as 
meaning <i>show me even more details</i> and as you can see in the following we now
see the individual OSTs associated with each file system.

<div class=terminal-wide14><pre>
$ collectl -sL -oTm
# LUSTRE CLIENT DETAIL
#            Filsys  Ost      Reads ReadKB  Writes WriteKB
15:19:17.007 spfs1  OST0000      0      0       0       0
15:19:17.007 spfs1  OST0001      0      0       0       0
15:19:17.007 spfs2  OST0000      0      0       0       0
15:19:17.007 spfs2  OST0001      0      0       0       0
</pre></div>

So what else can we look at?  Lots! Lustre tracks <i>readahead</i> data which you can
show in brief format like this by using the <i>R</i> option noting this time we're 
also requesting date/time stamps be included.  Here we see the cache hits/misses
added to the brief display for the client.

<div class=terminal-wide14><pre>
$ collectl -s l -OR -oD
#                  <-------------Lustre Client-------------->
#Date    Time       Reads KBRead Writes KBWrite   Hits Misses
20080319 15:20:12       0      0      0       0      0      0
20080319 15:20:13       0      0      0       0      0      0
</pre></div>

or you can look at it in verbose format like this:

<div class=terminal-wide14><pre>
$ collectl -s l -OR --verbose
# LUSTRE CLIENT SUMMARY: READAHEAD
# Reads ReadKB  Writes WriteKB  Pend  Hits Misses NotCon MisWin LckFal  Discrd ZFile ZerWin RA2Eof HitMax
      0      0       0       0     0     0      0      0      0      0      0      0      0      0      0
      0      0       0       0     0     0      0      0      0      0      0      0      0      0      0
</pre></div>

Lustre also tracks client metadata, which you can request by specifying <i>-OM</i> and 
BRW stats which are selected by <i>-OB</i>.  However both are so wide they only 
have a verbose form and the following shows both being displayed at the same time.

<div class=terminal-wide14><pre>
$ collectl -s l -OBM
# LUSTRE CLIENT SUMMARY: RPC-BUFFERS (pages) METADATA
#Rds  RdK   1P   2P   4P   8P  16P  32P  64P 128P 256P Wrts WrtK   1P   2P   4P   8P  16P  32P  64P 128P 256P
   0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
# Reads ReadKB  Writes WriteKB  Open Close GAttr SAttr  Seek Fsynk DrtHit DrtMis
      0      0       0       0     0     0     0     0     0     0      0      0
</pre></div>

And if that's not enough, these even have a detailed mode of display and you can look 
at them by filesystem or OST.  You can specify any combinations of B,M and R with -O
to show the associated data in both summary or detail formats, though only <i>readahead</i>
appears in the brief display.  All others will force verbose format.

<p><b>Object Storage Server</b><p>
If you haven't figured it out yet, it's also possible to display OST level information
on an OSS by simply using the uppercase subsystem specification as you can see here,
noting just to be different I've chosen a second form of the date/timestamp.

<div class=terminal-wide14><pre>
$ collectl -sL -od
# LUSTRE FILESYSTEM SINGLE OST STATISTICS
#              Ost            Read Ops   Read KB      Write Ops   Write KB
03/19 15:32:14 spfs1-OST0000        0         0              0          0
03/19 15:32:14 spfs2-OST0000        0         0              0          0
03/19 15:32:15 spfs1-OST0000        0         0              0          0
03/19 15:32:15 spfs2-OST0000        0         0              0          0
</pre></div>

You can also show BRW stats as both summary and detail data as well and since they
look identical to the way they're displayed for a client, I won't bother repeating 
those forms here.

<p><b>Metadata Server</b><p>
As mentioned earlier, there is no detail data for an MDS nor are there any other
types of data other than that which can be displayed in summary mode.
<p>
<h3>Summary</h3> 
As you have seen, there is a wealth of data here and often you may not even know what
you want to look for.  When such is the case, just collect as much as you can and save
it in a file.  Then you can play it back later and display it in multiple formats.
<p>
I just want to close with an example of a problem with readaheads and Lustre 1.4
which demonstrates the power of a tool
like collectl that can show multiple types of data at once because I know a lot of 
people may still not be convinced.  Consider the following sample which was collected while
doing random 32KB reads of a large file.  See anything wrong?  Do you know why the
network bandwidth is so much higher than the lustre client read rate?  How long would it
have taken to even realize there is a problem?

<div class=terminal-wide14><pre>
$ collectl -snl -oT
#         <----------Network----------><-------------Lustre Client-------------->
#Time     netKBi pkt-in netKBo pkt-out  Reads KBRead Writes KBWrite   Hits Misses
08:14:18   41776  28310   1065   14786     50    200      0       0     30     20
08:14:19   38328  25987   1032   14078     62    248      0       0     35     19
08:14:20   44763  30337   1167   16114     58    232      0       0     30     20
08:14:21   43666  29596   1137   15632     46    184      0       0     30     16
08:14:22   33777  22905    891   12191     58    232      0       0     35     23
</pre></div>

It turned out the old algorithm triggered a readahead after 2 consecutive pages were read
and every 32KB read (which is 8 pages)
resulted in 1MB being read over the network, loaded in to cache and was then discarded!  
If the value of <i>max_read_ahead_mb</i> is set to 0 for the associated filesystem on
<i>each</i> client involved, you see 2 things - the network rate drops to track the 
client read rate and the <i>misses</i> goes to zero.

<div class=terminal-wide14><pre>
$ collectl -snl -oT
#         <----------Network----------><-------------Lustre Client-------------->
#Time     netKBi pkt-in netKBo pkt-out  Reads KBRead Writes KBWrite   Hits Misses
08:21:22       0      3      0       3     59    236      0       0      0      0
08:21:23     317    335     58     298     91    364      0       0      0      0
08:21:24     442    457     77     388    107    428      0       0      0      0
08:21:25     442    457     77     383     97    388      0       0      0      0
08:21:26     432    446     75     373     89    356      0       0      0      0
</pre></div>

<i>Caution - changing the value of the readable variable in /proc to 0 will eliminate
all readahead for </i>all readers<i> meaning any applications that do sequential
reads could have significant performance problems.  In other words, this is <i>not</i> a
recommendation to turn off readahead but rather to be aware of what it does and if you do
choose to turn it off to be aware of the consequences</i>
<p>
<i><b>Note: This readahead algorithm has changed with V1.6 and lustre no longer triggers readahead
on the third page read but rather on the third sequential read.</b></i>


</body>
</html>
